Skip to content

Exclude benchmark setup duration using iter_batched #2233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
cijothomas merged 2 commits into from
Oct 25, 2024
Merged

Exclude benchmark setup duration using iter_batched #2233

cijothomas merged 2 commits into from
Oct 25, 2024

Conversation

@clnv
Copy link
Contributor

@clnv clnv commented Oct 24, 2024 •
edited
Loading

Fixes #1805
Design discussion issue (if applicable) #

Changes

Use iter_batched for benchmarking Counter_Add_Sorted and Counter_Add_Unsorted to exclude the setup duration from the benchmark results.

With LargeInput

Counter_Created
Counter_Add_Sorted      time:   [170.49 ns 170.63 ns 170.89 ns]
                        change: [-11.149% -10.899% -10.583%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

Counter_Created
Counter_Add_Unsorted    time:   [172.93 ns 172.97 ns 173.01 ns]
                        change: [-11.878% -11.673% -11.426%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  5 (5.00%) high mild
  5 (5.00%) high severe

With SmallInput

Counter_Created
Counter_Add_Sorted      time:   [167.97 ns 168.04 ns 168.11 ns]
                        change: [-2.0473% -1.7160% -1.4582%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  4 (4.00%) high severe

Counter_Created
Counter_Add_Unsorted    time:   [171.76 ns 171.94 ns 172.18 ns]
                        change: [-1.0523% -0.8236% -0.6261%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe

OS: Ubuntu 24.04 (6.8.0-1014-azure)
Hardware: AMD EPYC 7763 64-Core (16) @ 2.44 GHz
RAM: 64.0 GB

Merge requirement checklist

  • CONTRIBUTING guidelines followed
  • Unit tests added/updated (if applicable)
  • Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
  • Changes in public API reviewed (if applicable)

@clnv clnv requested a review from a team as a code owner October 24, 2024 04:21
@clnv clnv force-pushed the perf/bench/iter-custom branch from b5a7d97 to f2417ae Compare October 24, 2024 04:22
@codecov
Copy link

codecov bot commented Oct 24, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.5%. Comparing base (a18853e) to head (39383a3).
Report is 1 commits behind head on main.

Additional details and impacted files 
@@          Coverage Diff          @@
##            main   #2233   +/-   ##
=====================================
  Coverage   79.5%   79.5%           
=====================================
  Files        121     121           
  Lines      21116   21116           
=====================================
+ Hits       16803   16805    +2     
+ Misses      4313    4311    -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cijothomas
cijothomas reviewed Oct 24, 2024
View reviewed changes
cijothomas
cijothomas approved these changes Oct 24, 2024
View reviewed changes
Copy link
Member

@cijothomas cijothomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!
Would love to see more contributions!

utpilla
utpilla reviewed Oct 24, 2024
View reviewed changes
@clnv clnv force-pushed the perf/bench/iter-custom branch from f2417ae to 39383a3 Compare October 24, 2024 22:11
@cijothomas cijothomas merged commit 5628f66 into open-telemetry:main Oct 25, 2024
24 of 25 checks passed
@clnv clnv deleted the perf/bench/iter-custom branch October 25, 2024 04:35
@utpilla utpilla mentioned this pull request Jan 8, 2025
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lalitb lalitb lalitb left review comments

@utpilla utpilla utpilla approved these changes

@cijothomas cijothomas cijothomas approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Avoid random generation cost from all perf tests

4 participants

@clnv @lalitb @cijothomas @utpilla